fix(#1433): clear error for leading-underscore attribute names#1441
Merged
dimitri-yatsenko merged 2 commits intomasterfrom Apr 29, 2026
Merged
Conversation
The attribute_name parser in declare.py was pp.Word over [a-z] init chars
and [a-z0-9_] body chars, rejecting any name starting with `_`. But the
framework already treats names starting with `_` as hidden attributes
(Heading.attributes filters by is_hidden = name.startswith("_")), and
internal hidden columns like _job_start_time, _job_duration, _job_version,
and _singleton are injected programmatically, bypassing the parser.
User-defined hidden attributes — documented at
docs.datajoint.com/reference/specs/table-declaration/#34-hidden-attributes —
hit the parser and failed with a cryptic pyparsing ParseException.
Allow `_` in the init-chars set so user code like
_params_hash: varchar(32)
unique index (tool, _params_hash)
declares cleanly. Names starting with a digit are still rejected.
Fixes #1433
2 tasks
User-defined hidden attributes (names starting with `_`) are intentionally
not supported. The framework filters hidden columns out of every public
API surface — fetch, dict restriction, insert, update1, describe — and
populates platform-managed hidden columns (`_job_*`, `_singleton`) via
raw SQL during the populate() lifecycle, not via the user-facing methods.
Allowing users to declare hidden columns produces a feature with no
public-API write path, no describe() round-trip, and silent dict-
restriction filtering. The right fix for cases users reach for hidden
attributes (e.g. an index-backing hash like `params_hash`) is a regular
attribute.
Add a pre-flight check in compile_attribute that detects a leading
underscore and raises DataJointError with a clear message pointing to
the alternative, instead of leaking pyparsing internals:
Attribute name in line "_hidden: bool" starts with an underscore.
Names with leading underscore are reserved for platform-managed
columns (e.g. _job_start_time, _singleton). Use a regular attribute
name; if you need to control visibility at the call site, use proj().
Platform code is unaffected: `_job_*` and `_singleton` are injected
programmatically *after* parsing, so they bypass compile_attribute.
Replaces 7 unit tests asserting "parser accepts _" with 4 asserting
"compile_attribute rejects _ with helpful message" and "parser remains
strict".
Fixes #1433
dimitri-yatsenko
added a commit
to datajoint/datajoint-docs
that referenced
this pull request
Apr 29, 2026
Updated to reflect the design decision in datajoint/datajoint-python#1441: the parser keeps rejecting leading-underscore attribute names and now returns a clear DataJointError instead of a cryptic ParseException. Reframe §3.4 around the platform-managed-only intent: - Lead paragraph states up-front that user-defined hidden attributes are not supported, and shows the new error message users will see. - Drop the "User-defined hidden attributes" subsection and the _params_hash hidden example. - Keep the platform-attributes table and the behavior matrix — both are still useful for users encountering platform-managed hidden columns (_job_start_time, etc.) in fetch results, joins, and describe output. - Add an explanation paragraph ("Why users can't declare them") covering the no-write-path / no-round-trip / silent-filter rationale. - Replace the user-defined example with a regular-attribute example (params_hash backing a unique index), demonstrating the recommended pattern: declare as a regular attribute, use proj() at the call site for visibility control.
kushalbakshi
approved these changes
Apr 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #1433 by replacing the cryptic `pyparsing.ParseException` that users hit when declaring `_hidden: bool` with a clear `DataJointError` that explains the constraint and points to the alternative.
Design decision: don't allow user-defined hidden attributes
Earlier drafts of this PR loosened the parser to accept leading underscores, on the assumption that users should be able to declare hidden attributes the same way the framework does. On reflection, that's wrong:
So this PR keeps the parser strict and improves the error message instead.
Fix
Pre-flight check in `compile_attribute` (declare.py:858) that detects a leading underscore and raises a `DataJointError` before the parser is invoked:
```
Attribute name in line "_hidden: bool" starts with an underscore.
Names with leading underscore are reserved for platform-managed
columns (e.g. _job_start_time, _singleton). Use a regular attribute
name; if you need to control visibility at the call site, use proj().
```
The parser regex is unchanged from master. Platform code is unaffected because `job*` and `_singleton` bypass `compile_attribute` entirely.
Companion docs PR
datajoint/datajoint-docs#162 — reworked to reflect the new design. The §3.4 section is now framed as platform-only, the `_params_hash` user-defined example is removed, and the table of behaviors is preserved as a reference for the platform-managed columns.
Test plan